On-chip image sensor data compression

ABSTRACT

An image sensor compresses image data prior to transmitting the image data to a DSP. The image sensor captures light representing an image, for instance via a camera&#39;s aperture. A focal plane array converts the captured light into pixel data. The pixel data is sorted into categories, and is compressed in parallel by a compression engine. The compressed pixel data is then sent to a DSP, which may be located off-chip. The DSP then decompresses the compressed pixel data, performs image signal processing operations on the compressed pixel data, and then compresses the processed pixel data into a digital image format. The image sensor may buffer the pixel data for one or more images to accommodate for slowdown by the compression engine. The pixel data may be sorted by row and column of a pixel array. Alternatively, the pixel data may be sorted by color from a Bayer color filter.

BACKGROUND

1. Technical Field

This disclosure relates to image capturing, and more specifically, tothe compression of image data in an image sensor chip, prior totransmission to a digital signal processor (DSP).

2. Description of the Related Arts

The advancement of digital video and image encoding has led toincreasingly sophisticated image capture techniques at increasingly highresolutions and frame rates. For instance, common media formats, such asBlu-Ray discs, internet video, and cable/satellite television, are ableto display content at a 1080P resolution (1920×1080 pixels progressivelyscanned) at 60 frames per second (“fps”). Certain displays are able todisplay resolutions of 2560×2048 pixels or 3260×1830 pixels or higher atframe rates of 120 frames per second or higher. As encoding and displaytechnology advances, frame rates and resolutions will increaseaccordingly.

The capture of digital images by an image capture device (hereinafter“camera”) is performed by an image sensor. Many types of image sensorsare commonly used in cameras and other image-capturing devices, such ascharge-coupled devices (CCDs) and complementarymetal-oxide-semiconductors (CMOSs). Image sensors convert light, such aslight entering the aperture of a camera through a camera lens, intoimage information. In this way, a camera can “capture” objects before itby converting the light reflected from the objects and passing throughthe camera lens into an image.

Image data from image sensors must often be processed into a particularimage format prior to the image being rendered by a display. Image datais typically processed by a DSP, which is often located off-chip fromthe image sensor. Image sensors and DSPs are typically connected bybuses coupled to the image sensors' pins and the DSPs' pins. As imageresolution and frame rate increase, the amount of image data produced bythe image sensor increases, along with the amount of power required toaccommodate the transfer of image data from the image sensor to the DSP.In some circumstances, the amount of power required to accommodate sucha transfer of image data rapidly depletes the battery life of a camera,or exceeds the amount of power available to the camera. In addition,increasing the power consumption in transferring image datacorrelatively increases the noise (such as the electromagnetic noise)impacting the image data.

The limited ability to transfer image data between an image sensor and aDSP thus creates a bottleneck in the image capture process. One solutionis to increase the number of pins on the image sensor and DSP toincrease the bus bandwidth between the image sensor and DSP. Such asolution, however, may not be physically possible due to the limitedreal estate of many image sensor chips and DSPs. Alternatively, imagesor video may be captured at a lower resolution and at a lower frame rateto reduce the power-limiting bandwidth requirements between the imagesensor and the DSP. However, this solution results in a lower qualityimage or video. Thus, camera makers are increasingly required to balancethe competing requirements of image and video quality with powerconsumption and chip real estate.

BRIEF DESCRIPTIONS OF THE DRAWINGS

The disclosed embodiments have other advantages and features which willbe more readily apparent from the following detailed description of theinvention and the appended claims, when taken in conjunction with theaccompanying drawings, in which:

FIG. 1 is a block diagram illustrating an embodiment of an image capturesystem.

FIG. 2 is a block diagram illustrating an embodiment of an image captureand display system environment.

FIG. 3 is a block diagram illustrating an embodiment of an image sensorchip.

FIG. 4 is a block diagram illustrating an embodiment of an image sensorchip.

FIG. 5 illustrates an embodiment of a process for compressing pixel dataprior to transmitting the data to a DSP.

DETAILED DESCRIPTION

The Figures and the following description relate to preferredembodiments by way of illustration only. It should be noted that fromthe following discussion, alternative embodiments of the structures andmethods disclosed herein will be readily recognized as viablealternatives that may be employed without departing from the principlesof what is claimed.

Reference will now be made in detail to several embodiments, examples ofwhich are illustrated in the accompanying figures. It is noted thatwherever practicable similar or like reference numbers may be used inthe figures and may indicate similar or like functionality. The figuresdepict embodiments of the disclosed system (or method) for purposes ofillustration only. One skilled in the art will readily recognize fromthe following description that alternative embodiments of the structuresand methods illustrated herein may be employed without departing fromthe principles described herein.

Configuration Overview

An image capture system provides photographers, professional andamateur, a configuration for capturing images and video at resolutionsand frame rates which previously required a performance-limiting amountof power for many camera configurations. The cameras described hereinmay be consumer grade while the configuration of the cameras allows forthe transfer of image data between an image sensor and a DSP at highresolutions and frame rates without the expense of professional gradeequipment.

The image capturing system described herein could be used to allowconsumers to capture high-resolution and high-frame rate images andvideo of local events or activities, including sporting events, theaterperformances, concerts, weddings, or other events without requiring aprofessional photographer and advanced camera equipment. It should benoted that for the purposes described herein, video capture is performedby the capture of successive images (“frames”), and thus functionalitydescribed herein making reference to one of either video capture orimage capture is applicable to both.

Image Capture System Overview

FIG. 1 is a block diagram illustrating an example embodiment of an imagecapture system. The image capture system includes an image sensor chip100 and a DSP chip 150, each of which may be contained within a camera.Light 110 enters the image sensor chip 100 through, for example, acamera lens and aperture, and is converted into compressed pixel data.The compressed pixel data is sent to the DSP chip 150, where it isconverted into a formatted image and output as the image 190.

The light 110 is received by the image sensor chip 100 at alight-to-signal conversion module 120. The light-to-signal conversionmodule 120 converts the light into an electrical signal that is receivedby a focal plane array (FPA) 130. In one embodiment, the light-to-signalconversion module converts the received light 110 into voltagesrepresenting properties of the received light 110, for examplefrequency, wavelength, intensity, magnitude, and the like. The FPA 130converts the electrical signals from the light-to-signal conversionmodule 120 into pixel data. In one embodiment, the FPA 130 outputs pixeldata sequentially for each frame of video captured by the image sensorchip 100. It should be noted that although the functionality of thelight-to-signal conversion module 120 and the FPA 130 are discussedseparately herein, the FPA 130 may perform the functionality of the bothmodules.

The pixel data produced by the FPA 130 may be spatial in form. Forinstance, the pixel data may include a description of image data foreach of a plurality of image locations (“pixels”) within (for example) arectangle of light (the “pixel array”) captured by the image sensor (the“image”). The pixel data may include, for each pixel in the image, anaddress within the image representing the location of the pixel in theimage, such as an x-coordinate and y-coordinate or other unique locationidentifier. Pixel data may also include, for each pixel, colorinformation describing the pixel, grey-scale or monochrome informationdescribing the pixel, luminance information describing the pixel, or anyother information related to the pixel specifically or the display ofpixels in an image generally.

The FPA 130 outputs the pixel data to the compression engine 140. Thecompression engine 140 performs compression on the received pixel data.Any type of compression may be performed by the compression engine 140,particularly compressions known in the art to be beneficial in imageand/or video compression. For example, the pixel data may be compressedusing a wavelet transform or a discrete cosine transform (DCT), whichmay be followed by quantization and entropy encoding. The pixel data maybe compressed using run-length encoding, predictive encoding, entropyencoding, adaptive dictionary algorithms, deflation, or other methods oflossless image compression. The pixel data may be compressed using colorspace reduction, chroma subsampling, transform codings other thanwavelet transforms and DCTs, fractal compression, or other methods oflossy image compression. The pixel data may be compressed using motioncompensation, such as global motion compensation, block motioncompensation, variable block-size motion compensation, overlapping blockmotion compensation, or other methods of motion compensation. The pixeldata may be compressed using 3D coding, for instance by color shifting,pixel subsampling, and enhanced video stream coding. In one embodiment,the compression engine 140 may compress data up to 3 or 4 times or more.It should be noted that the inclusion of the compression engine 140 onthe image sensor chip 100 in some embodiments may increase fabricationand production costs or the image sensor chip 100, and may increase thesize of the image sensor chip 100.

In one embodiment, the FPA 130 may be modified to include logicduplicated along the rows or columns of pixels within the image sensorchip 100; such logic may be configured to perform wavelet transformoperations or other compression operations. For example, in anembodiment where multiple wavelet transforms are performed on pixeldata, wavelet transform logic within the FPA 130 may perform a firstwavelet transform on the pixel data, and the compression engine 140 mayperform additional wavelet transforms on the pixel data. By includingcompression logic within the rows and columns of the pixels of the imagesensor chip 100, the geometry of the image sensor can be beneficiallyutilized to improve the compression performance of the on-image sensorchip image compression. For instance, wavelet transforms may beperformed on columns or rows of pixels, and utilizing the column/rowconfiguration of the pixels of the image sensor chip 100 allows wavelettransforms to be performed without requiring areconfiguration/re-ordering of pixel data.

The amount of compression performed by the compression engine 140 may bevariable and/or situational. In one embodiment, if the output busbetween the image sensor chip 100 and the DSP chip 150 isbandwidth-limited, the compression engine 140 may increase thecompression on the received pixel data as needed to ensure that thecompressed pixel data output by the compression engine 140 does notexceed the maximum bandwidth of an output bus of the image sensor chip.For example, if the image sensor chip 100 is capturing video resultingin 10 MB of pixel data per frame at 30 frames per second fps, and if theimage sensor chip's output bus is bandwidth limited to 150 MB/s, thecompression engine 140 may compress the received pixel data by a factorof 2 to reduce the total amount of data transmitted from the imagesensor chip 100 to the DSP chip 150 from 300 MB to 150 MB. As the amountof pixel data and the frame rate of video increase and decrease, theamount of compression performed by the compression engine 140 mayincrease or decrease proportionately.

Similarly, the amount of compression performed by the compression engine140 may increase or decrease based on the amount of power available tooutput pixel data to the DSP chip 150, for instance based on apower-savings setting (such as a “conserve power” mode set by a user ona camera). The amount of compression performed by the compression engine140 may increase as the amount of power available to output pixel datadecreases, and vice versa. Beneficially, the amount of power used tocompress pixel data by the compression engine 140 may be less than theamount of power required to output uncompressed pixel data to the DSPchip 150, resulting in a more power-efficient solution to transfer pixeldata from the image sensor chip 100 to the DSP chip 150.

The FPA 130 may output portions of the pixel data to the compressionengine 140 in parallel, allowing the compression engine 140 to takeadvantage of properties of data parallelism when compressing thereceived data. For instance, the FPA 130 may output four lines of pixeldata to the compression engine 140, allowing the compression engine 140to compress four components of pixel data simultaneously. It should beemphasized that the FPA 130 may output the pixel data to the compressionengine 140 serially, allowing the compression engine 140 to compress thepixel data serially, and not in parallel.

In order to compress the pixel data in parallel, the pixel data may beorganized into pixel data categories. Pixel categories are groups ofpixels with one or more common properties, characteristics, orqualities. In one embodiment, pixels are organized into pixel categoriesfor the purposes of aiding or increasing the efficiency of parallelpixel compression. Examples of pixel data categories includeorganization by color, by color plane (such as Red, Green1, Blue, andGreen2), by luminosity, by chrominance, by location or spatiality (forinstance by even or odd column and row, by x and y coordinates, or byimage quadrant), by entropy, by similarity to nearby pixels, by index,or by any other means of organizing or categorizing pixels.

The FPA 130 or compression engine 140 may additionally include one ormore buffers, allowing the pixel data of multiple images to be bufferedor stored in buffer memory simultaneously. Alternatively, the buffersmay be coupled to but independent of the FPA 130 and the compressionengine 140. The buffers may allow for slowdown or variable compressionperformance by the compression engine 140 without detrimentallyaffecting the overall performance of the image sensor chip 100. In oneembodiment, the buffers include pixel row buffers, for instance 6-8lines deep. In this embodiment, in addition to buffering pixel data, thebuffers may be used by the compression engine 140 to store intermediarydata when compressing the pixel data. For example, if the compressionengine 140 is performing a wavelet transform on the pixel data, thecompression engine 140 may store wavelet transform coefficients or othercomponents of the wavelet transform in the buffer memory during theperformance of the wavelet transform.

The DSP chip 150 receives the compressed pixel data at the decompressionmodule 160. The decompression module 160 decompresses the receivedcompressed pixel data such that the original pixel data outputted by theFPA 130 is retrieved. In one embodiment, the decompression module 160performs the inverse of the compression performed by the compressionengine 140. Thus, by compressing and decompressing the pixel data outputby the FPA 130, the pixel data may be sent from the image sensor chip100 to the DSP chip 150 while reducing the amount of bandwidth (andaccordingly, power) that would otherwise be required to send such data.The decompression module 160 then outputs the retrieved pixel data tothe ISP module 170.

The decompression module 160 may also downscale the retrieved pixeldata, for instance during the decompression of the pixel data.Downscaling the retrieved pixel data may result in the lowering of theresolution of the pixel data. For example, if the pixel data captured bythe image sensor chip is captured at a resolution of 2720×1530 pixels,the decompression module 160 may downscale the pixel data by a factor of12, resulting in a downscaled pixel data resolution of 1920×1080 pixels.

The decompression module 160 may also perform certain image processingoperations. In one embodiment, the decompression module 160 performsconversion between various color mapping schemes. For example, thedecompression module 160 may convert between a Bayer scheme, an RGBscheme, an sRGB scheme, a CMYK scheme, and the like. Alternatively,these and other image processing operations may instead be performed bythe ISP 170.

The ISP 170 receives the decompressed (and optionally downscaled andprocessed) pixel data from the decompression module 160, performs imagesignal processing on the decompressed pixel data, and outputs theprocessed data to the final compression module 180. The ISP 170 mayperform any suitable image processing operation on the decompressionpixel data, such as resolution adjustment, color conversion, brightnessadjustment, pixel formatting adjustment, and the like. In oneembodiment, the ISP 170 performs image processing as required by thefinal compression module 180. For example, if the final compressionmodule 180 requires pixel data to be formatted in a particular way inorder to compress the pixel data into a particular image format, the ISP170 may accordingly format the pixel data as required by the finalcompression module 180.

The final compression module 180 receives the processed pixel data,formats the pixel data into a particular pixel data format, andcompresses the formatted pixel data into the image 190. It should benoted that the image 190 may refer to a particular frame of video, andthus the final compression module 180 may consecutively outputsuccessive images 190 in order to output video. The final compressionmodule 180 may output the image 190 in any suitable image and/or videoformat, such as the JPG format, PNG format, Bitmap format, GIF format,MOV format, AVI format, WMV format, H.264, MPEG format, raw image ormovie data formats, and the like.

System Architecture

FIG. 2 is a block diagram illustrating an embodiment of an image captureand display system environment. In the environment of FIG. 2, an imagecapture device 210, an external display 230, an external storage module240, a broadcast client 250, and a viewer client 260 all communicatethrough a connecting network 200. Other embodiments of such anenvironment may contain fewer or additional modules, which may performdifferent functionalities than described herein. Although only one ofeach component is illustrated in the environment of FIG. 2, otherembodiments can have any number of each type of component, such asthousands or millions.

The image capture device 210 and other components of FIG. 2 may beimplemented in computers adapted to execute computer program modules. Asused herein, the term “module” refers to computer-readable programinstructions and/or data for providing the specified functionality. Amodule can be implemented in hardware, firmware, and/or software. In oneembodiment, the modules are stored on one or more storage devices,loaded into memory, and executed by the processors. Storage devices,memory, and processors are described in greater detail in thedescription of the image capture device 210 below; this descriptionapplies equally to any of the components of FIG. 2.

The image capture device 210 of the embodiment of FIG. 2 includes theimage sensor chip 100, the DSP 150, a processor 212, memory 214, a localdisplay 216, a user input 218, a network adapter 220, and an internalstorage module 222. In other embodiment, the image capture device 210may include fewer or additional components not shown for the purposes ofsimplicity. For instance, the image capture device 210 may include aninternal bus, allowing the components of the image capture device 210 tocommunicate. In addition, not shown are common components of an imagecapture device 210, such as a lens, an aperture, a battery or otherpower supply, communication ports, speakers, a microphone, and the like.In one embodiment, the image sensor chip 100 and the DSP chip 150 arethe image sensor chip 100 and the DSP chip 150 of the embodiment of FIG.1.

The processor 212 may be any general-purpose processor. The processor212 is configured to execute instructions, for example, instructionscorresponding to the processes described herein. The memory 214 may be,for example, firmware, read-only memory (ROM), non-volatile randomaccess memory (NVRAM), and/or RAM. The internal storage module 222 is,in one embodiment, an integrated hard disk drive or solid state memorydevice, though in other embodiments may be a removable memory device,such as a writeable compact disk or DVD, a removable hard disk drive, ora removable solid state memory device. The memory 214 and/or theinternal storage module 222 are configured to store instructions anddata that can be executed by the processor 212 to perform the functionsrelated to the processes described herein. In one embodiment, thefunctionalities performed by the image sensor chip 100 or its componentsor the DSP chip 150 or its components are performed by the processor 212and instructions stored in the memory 214 and/or the internal storagemodule 222.

The local display 216 may be implemented with an integrated LCD screenor other similar screen, such as a monochromatic display or otherdisplay. Alternatively, the local display 216 may be implemented with aremovable display module, such as a LCD pack configured to couple to theimage capture device 210 and communicate with the components of theimage capture device 210 through an internal bus.

The local display 216 may be configured to operate as a user interfacefor a user of the image capture device 210. In one embodiment, the localdisplay 216 displays menus, HUDs, UIs, and the like to allow a user toutilize the functionalities of the image capture device 210 or to informthe user of information related to the image capture device 210, such asthe amount of available storage remaining, the amount of powerremaining, the current resolution and/or frame rate of the image capturedevice 210, and any other settings or information related to the imagecapture device 210. In one embodiment, the image capture device 210 isconfigured to perform the functions of an electronic view finder,allowing a user of the image capture device 210 to view the imagesand/or video that the image capture device 210 will capture or iscapturing responsive to a capturing action performed by a user of theimage capture device 210. In one embodiment, the local display 216 isconfigured to display previously captured images and videos.

The user input 218 comprises a solid state and/or mechanical interface.For example, the user input 218 may include one or more buttons on theexterior of the image capture device 210. The user input 218 isconfigured to receive a selection action from a user of the imagecapture device 210 and allows the user to interact with the imagecapture device 210. Alternatively, the user input 218 may include atouch-screen component of the local display 216, an external inputdevice, such as a keyboard, mouse or other controller configured tocommunicate with the image capture device 210 via an input port or thenetwork adapter 220, or any other means of interacting with the imagecapture device 210. A user may use the user input 218 to interact withthe image capture device 210 and perform functions of the image capturedevice 210, such as navigating through previously captured images orvideo, editing or deleting previously captured images or video, alteringthe image capture device's settings (such as the resolution or framerate of future captured images or video, adjusting a power-save mode, oradjusting other camera parameters or settings, such as a night/day mode,a self-timer, an exposure length, and the like), turning the imagecapture device 210 on and off, communicating with external modules viathe network adapter 220, and so forth.

The network adapter 220 communicatively couples the image capture device210 to external modules via the connecting network 200. The networkadapter 220 may include a network card, a modem, or any deviceconfigured to allow the image capture device 210 to communicate with theother components of FIG. 2 via the connecting network 200 and viceversa.

The connecting network 200 enables communications among the entitiesconnected to it. In one embodiment, the connecting network 200 is theinternet and uses standard communications technologies and/or protocols.Thus, the connecting network 200 can include links using technologiessuch as Ethernet, 802.11, worldwide interoperability for microwaveaccess (WiMAX), long term evolution (LTE), 3G, and the like. Similarly,the networking protocols used on the connecting network 200 can includemultiprotocol label switching (MPLS), the transmission controlprotocol/Internet protocol (TCP/IP), the User Datagram Protocol (UDP),the hypertext transport protocol (HTTP), the simple mail transferprotocol (SMTP), the file transfer protocol (FTP), and the like. Thedata exchanged over the connecting network 200 can be represented usingtechnologies and/or formats including the hypertext markup language(HTML), the extensible markup language (XML), and the like. At least aportion of the connecting network 200 can comprise a mobile (e.g.,cellular or wireless) data network such as those provided by wirelesscarriers. In some embodiments, the connecting network 200 comprises acombination of communication technologies.

The external display 230 may include any type of display configured todisplay images or videos captured by the image capture device 210, menusor other interfaces of the image capture device 210, or any othercontent from the image capture device 210. For example, the externaldisplay may be a television, a monitor, a mobile phone, and the like.The external display 230 may receive content from the image capturedevice 210 from the image capture device via the connecting network 200,or from the external storage module 240, the broadcast client 250, andthe like. In one embodiment, the image capture device 210 transmitscontent from the image capture device 210 to the broadcast client 250via the internet, and the broadcast client 250 broadcasts the receivedcontent to a mobile device comprising the external display 230 via acellular network.

The external storage module 240 is configured to receive and storecontent from the image capture device 210. In one embodiment, theexternal storage module 240 includes a database, a datacenter, anexternal hard disk drive, an external computer, and the like. Theexternal storage module 240 may further be configured to provide storedcontent to the components of the embodiment of FIG. 2. For instance, auser may retrieve previously stored images or video from the externalstorage module 240 for display on the image capture device 210 or theexternal display 230, or for broadcasting by the broadcast client 250.

The broadcast client 250 comprises a computer or other electronic deviceoperated by a user to create and control broadcasts of content from theimage capture device. For example, in one embodiment, the broadcastclient 250 includes a personal computer or mobile device executing a webbrowser that can retrieve content stored on the image capture device 210or the external storage module 240. The broadcast client 250 may thenbroadcast the retrieved content to other entities via the connectingnetwork 200.

The viewer client 260 is a computer or other electronic device used byone or more users to view broadcasts generated and served by thebroadcast client 250. For example, in one embodiment, the viewer client260 includes a personal computer or mobile device executing a webbrowser. A user can access the broadcast client 250 using the viewerclient 260, and can use the available tools to browse and/or search foravailable broadcasts, or view a selected broadcast. In one embodiment,the external display 230 and the viewer client 260 are implemented inthe same entity.

Operational Configurations

FIGS. 3 and 4 are block diagrams illustrating various embodiments of animage sensor chip. In other embodiments, the image sensor chip includesfewer, additional, or different components, which may perform differentfunctionalities. In the embodiments of FIG. 3 and FIG. 4, pixel data issorted into categories and compressed on the image sensor chip 100 inparallel; it should be emphasized that in other embodiments, the pixeldata may not be sorted into categories prior to compression, and/or isnot compressed in parallel.

In the embodiment of FIG. 3, the FPA 130 of the image sensor chip 100outputs the odd pixel rows of the captured pixel array to the odd rowsmodule 300 and outputs the even pixel rows of the captured pixel arrayto the even rows 310 module. The FPA 130 may output the odd and evenpixel rows on alternating clock cycles, may output portions of the oddpixel rows in sequential clock cycles followed by portions of the evenpixel rows in sequential clock cycles, or may output multiple or alleven and odd pixel rows for an image in a single clock cycle.

The odd rows module 300 and the even rows module 310 may be implementedin a pixel row buffer, capable of storing and buffering the pixel rowsof one or more captured images simultaneously. By buffering the odd andeven pixel data of more than one image, the odd rows module 300 and theeven rows module 310, respectively, are capable of compensating forslowdown by the compression engine. The performance of the compressionengine may not be constant from a time-interval perspective; compressingcertain images may take more time than the time it takes to transfer theimages from the FPA 130 to the odd rows module 300 and the even rowsmodule 310. Similarly, compressing certain images may take less timethan the time it takes to transfer the images from the FPA 130 to theodd rows module 300 and the even rows module 310. Accordingly, so longas the average time to compress image data is less than the average timeto transfer image data from the FPA 130, buffering the odd and evenpixel data of images in the odd rows module 300 and the even rows module310, respectively, allows the image sensor chip 100 to avoid aperformance setback due to the variable performance of the compressionengine.

The odd rows module 300 outputs odd column pixel data to the odd columnsmodule 320 and even column pixel data to the even columns module 330.Likewise, the even rows module 310 outputs odd column pixel data to theodd columns module 340 and even column pixel data to the even columnsmodule 350. In one embodiment, in a first clock cycle, the odd rowsmodule 300 receives odd rows pixel data for a first image from the FPA130. Continuing with this embodiment, in a second clock cycle, the oddrows module 300 outputs the odd and even column pixel data from the oddrow pixel data of the first image to the odd columns module 320 and theeven columns module 330, respectively, and the even rows module 310receives even row pixel data for the first image from the FPA 130. In athird clock cycle, the even rows module 310 outputs the odd and evencolumn pixel data from the even row pixel data of the first image to theodd columns module 340 and the even columns module 350, respectively,and the odd rows module 300 receives odd rows pixel data for a secondimage. This alternating clock cycle process may repeat for any number ofimages. It should be noted that in one embodiment, any number of clockcycles (such as one, two, or more than two) may be required for each ofthe odd rows module 300 and the even rows module 310 to both receiverows pixel data and output column pixel data.

The odd columns module 320, the even columns module 330, the odd columnsmodule 340, and the even columns module 350 (collectively, the columnsmodules 320-350) output column pixel data to the 4:1 compression engine360. Although a 4:1 compression engine 360 is illustrated in theembodiments of FIGS. 3 and 4, the compression engine may implement anyamount of compression, for instance 2:1, 8:1, 16:1, and the like. Thecolumns modules 320-350 may output column pixel data to the 4:1compression engine 360 simultaneously (for instance in a single clockcycle), or may output column pixel data as it is received by the columnsmodules 320-350.

Similarly to the odd rows module 300 and the even rows module 310, thecolumns modules 320-350 may be implemented in pixel column buffers. Inthis embodiment, the pixel column data of multiple images may be storedand buffered to account for performance slowdowns by the compressionengine. In the embodiment where either the odds rows module 300 and theeven rows module 310, or the columns modules 320-350, or both, areimplemented in buffers, the compression engine may request pixel datafor an image when the compression engine is inactive. In the embodimentof FIG. 3, the 4:1 compression engine 360 may request pixel data fromthe odd rows module 300 and even rows module 310, or from the columnsmodules 320-350 for an image when the 4:1 compression engine 360 hasfinished compressing a previous image, when the 4:1 compression engine360 anticipates that it will be finished compression a previous image,or any other time the compression engine 360 isn't busy or is otherwiseidle.

The 4:1 compression engine 360 receives column pixel data for an imagefrom the columns modules 320-350 and compresses the column pixel data ata 4:1 ratio. By receiving 4 distinct feeds of column pixel data, the 4:1compression engine 360 may take advantage of the parallelism in thecolumn pixel data in compression the data. In one embodiment, the 4:1compression engine 360 compresses the pixel data for each imageindividually; in other embodiments, the 4:1 compression engine 360compresses data incrementally across multiple sequential images. The 4:1compression engine 360 outputs the compressed pixel data as compresseddata 370.

In the embodiment of FIG. 4, the FPA 130 includes a Bayer color filter,and converts image data into a RGBG (Red, Green1, Blue, Green2) colorpixel array. In this embodiment, the FPA 130 outputs the red pixel datafor an image to the red module 400, outputs a first portion of the greenpixel data for the image to the green1 module 410, outputs the bluepixel data for the image to the blue module 420, and outputs a secondportion of the green pixel data for the image to the green2 module 430.The FPA 130 may output color pixel data for an image in a single clockcycle, or may output portions of the color pixel data in different oracross multiple clock cycles.

The red module 400, the green1 module 410, the blue module 420, and thegreen2 module 430 output the color pixel data to the 4:1 compressionengine 440. Similarly to the embodiment of FIG. 3, the red module 400,the green1 module 410, the blue module 420, and the green2 module 430may be implemented in pixel buffers, compensating for slowdown by the4:1 compression engine 440. Similar to the 4:1 compression engine 340,the 4:1 compression engine 440 may take advantage of the parallelismacross the four color pixel data feeds. The 4:1 compression engine 440compresses the color pixel data at a 4:1 ratio and outputs thecompressed pixel data as the compressed data 450. It should be notedthat the processes described in conjunction with FIGS. 3 and 4 may beembodied as instructions that may be executed by the processor 212, thememory 214, and the internal storage module 222 of the image capturedevice 210.

A variation of the compression described in the embodiment of FIG. 4 mayalso be utilized. In this variation, chroma channel differencing isemployed prior to the compression of pixel data. For example, the fourinputs to the 4:1 compression engine 440 may include the pixel dataquantities (Green1+Green2), (Green1−Green2), (2*Red−Green1−Green2), and(2*Blue−Green1−Green2). The 4:1 compression engine 440 may compressthese pixel data quantities as described above. This form of pixel datacompression is described in greater detail in U.S. Pat. No. 8,014,597,the contents of which are hereby incorporated in their entirety.

FIG. 5 illustrates an embodiment of a process for compressing pixel dataprior to transmitting the pixel data to a DSP. Light received by, forinstance, a camera is converted 500 into pixel data. The pixel data issorted 510 into categories, for instance by an FPA. For example, thepixel data may be sorted by column and row for a pixel array, or bycolor in a Bayer color filter. The categorized pixel data is compressed520 in parallel. In one embodiment, each category of pixel data for animage is received in a distinct pixel data feed, and all categories ofpixel data are received in parallel. The compression of the categorizedpixel data may take into account parallelism in the pixel data, whichmay improve compression performance. In one embodiment, the categorizedpixel data for multiple images is buffered, compensating for compressionperformance variances without affecting overall image captureperformance.

The compressed pixel data may be transmitted 530 to a DSP. Thecompressed pixel data is decompressed 540. In one embodiment, thecompressed pixel data may also be downscaled, and/or various imageprocessing operations may be performed on the compressed pixel data,either before or after the compressed pixel data is decompressed. Imagesignal processing is performed 550 on the decompressed pixel data, andthe processed pixel data is compressed into a digital image format 560.It should be emphasized that the process of the embodiment of FIG. 5 maybe performed on image sequences, such as videos; in such an embodiment,each step in the process of FIG. 5 may be performed for each image inthe image sequence, and the processed pixel data for the image sequencemay be compressed into a digital movie format.

Compressing pixel data on an image sensor chip, prior to transmittingthe pixel data to the DSP, allows for an image capture system whichrequires fewer pins and transmission lines between the image sensor chipand the DSP. Further, less power is required to send the compressedpixel data to the DSP; as a result, less noise is introduced into thepixel data as a result of overheating. Finally, the amount of bandwidthrequired to transmit the compressed pixel data is reduced, allowing forhigher-resolution and higher-frame rate images and video to be capturedand transmitted off the image sensor chip to the DSP.

It is noted that terms “comprises,” “comprising,” “includes,”“including,” “has,” “having” or any other variation thereof, areintended to cover a non-exclusive inclusion. For example, a process,method, article, or apparatus that comprises a list of elements is notnecessarily limited to only those elements but may include otherelements not expressly listed or inherent to such process, method,article, or apparatus. Further, unless expressly stated to the contrary,“or” refers to an inclusive or and not to an exclusive or. For example,a condition A or B is satisfied by any one of the following: A is true(or present) and B is false (or not present), A is false (or notpresent) and B is true (or present), and both A and B are true (orpresent).

In addition, use of the “a” or “an” are employed to describe elementsand components of the embodiments herein. This is done merely forconvenience and to give a general sense of the invention. Thisdescription should be read to include one or at least one and thesingular also includes the plural unless it is obvious that it is meantotherwise.

Finally, as used herein any reference to “one embodiment” or “anembodiment” means that a particular element, feature, structure, orcharacteristic described in connection with the embodiment is includedin at least one embodiment. The appearances of the phrase “in oneembodiment” in various places in the specification are not necessarilyall referring to the same embodiment.

Upon reading this disclosure, those of skill in the art will appreciatestill additional alternative structural and functional designs for abroadcast management system as disclosed from the principles herein.Thus, while particular embodiments and applications have beenillustrated and described, it is to be understood that the disclosedembodiments are not limited to the precise construction and componentsdisclosed herein. Various modifications, changes and variations, whichwill be apparent to those skilled in the art, may be made in thearrangement, operation and details of the method and apparatus disclosedherein without departing from the spirit and scope defined in theappended claims.

What is claimed is:
 1. A method for compressing image data, the method comprising: receiving, by an image sensor chip, pixel data from a focal plane array at a first data rate, the pixel data comprising one or more captured images; sorting, by the image sensor chip, the received pixel data into a plurality of categories; selecting, by the image sensor chip, a first data compression factor based on the first data rate, the selected first data compression factor representing a magnitude of compression; compressing, by the image sensor chip, the categorized pixel data in parallel based on the selected first data compression factor; transmitting, by the image sensor chip, the compressed pixel data to a digital signal processor (DSP); decompressing the compressed pixel data at the DSP; and compressing the decompressed pixel into a digital image format.
 2. The method of claim 1, further comprising: capturing light with an image sensor chip; converting the captured light into pixel data by an focal plane array (FPA); and performing one or more image signal processing operations on the decompressed pixel data.
 3. The method of claim 1, wherein sorting the received pixel data into categories comprises sorting the received pixel data into odd row pixel data and even row pixel data, and further sorting the received pixel data into odd column and even column odd row pixel data and odd column and even column even row pixel data.
 4. The method of claim 1, wherein sorting the received pixel data into categories comprises sorting the received pixel data into color categories with a Bayer color filter, the color categories comprising red pixel data, blue pixel data, a first portion of green pixel data, and a second portion of green pixel data.
 5. The method of claim 1, further comprising: downscaling the decompressed pixel data into a resolution lower than the resolution of the captured image.
 6. A system for compressing image data, the system comprising: an image sensor chip comprising: an image sensor configured to capture light, a focal plane array configured to convert the captured light into pixel data at a first data rate, the pixel data representative of one or more captured images, a pixel sorting module configured to sort pixel data into categories, and a compression engine configured to select a data compression factor based on the first data rate and representing a magnitude of compression, to compress the categorized pixel data in parallel based on the selected data compression factor, and to transmit the compressed pixel data to a DSP, and a DSP comprising: a decompression engine configured to decompress the compressed pixel data, and a compression engine configured to compress the decompressed pixel data into a digital image format.
 7. The system of claim 6, wherein the DSP further comprises: an image signal processor configured to perform one or more image signal processing operations on the decompressed pixel data; and a downscaling engine configured to downscale the decompressed pixel data into a resolution lower than the resolution of the captured image.
 8. The system of claim 6, wherein the pixel sorting module is configured to sort the pixel data into odd row pixel data and even row pixel data, and wherein the pixel sorting module is further configured to sort the pixel data into odd column and even column odd row pixel data and odd column and even column even row pixel data.
 9. The system of claim 6, wherein the pixel sorting module is configured to sort the pixel data into color categories with a Bayer color filter, the color categories comprising red pixel data, blue pixel data, a first portion of green pixel data, and a second portion of green pixel data.
 10. A method for compressing image data, the method comprising: capturing light with an image sensor chip, the captured light comprising one or more captured images; converting, by an FPA, the captured light into pixel data at a first data rate; sorting the pixel data into a plurality of pixel data categories; selecting a data compression factor based on the first data rate, the selected data compression factor representing a magnitude of compression; and compressing, by the image sensor chip, the categorized pixel data in parallel based on the selected data compression factor prior to sending the pixel data to a DSP.
 11. The method of claim 10, wherein sorting the pixel data into a plurality of categories comprises sorting the pixel data into odd row pixel data and even row pixel data, and further sorting the pixel data into odd column and even column odd row pixel data and odd column and even column even row pixel data.
 12. The method of claim 10, wherein sorting the pixel data into a plurality of categories comprises sorting the pixel data into color categories with a Bayer color filter, the color categories comprising red pixel data, blue pixel data, a first portion of green pixel data, and a second portion of green pixel data.
 13. The method of claim 10, further comprising: buffering pixel data associated with one or more captured image prior to compressing the categorized pixel data in parallel.
 14. The method of claim 13, wherein pixel data associated with each of the plurality of pixel data categories is buffered in a buffer associated with the pixel data category.
 15. An image sensor for compressing image data, the image sensor comprising: a light-to-signal converter configured to capture light and to convert the captured light into an electronic signal, the captured light comprising a captured image; an FPA configured to convert the electronic signal into pixel data at a first data rate, and further configured to sort the pixel data into a plurality of pixel data categories; and a compression engine configured to select a data compression factor based on the first data rate and representing a magnitude of compression, and to compress the categorized pixel data in parallel based on the selected data compression factor prior to sending the pixel data to a DSP.
 16. The image sensor of claim 15, wherein the FPA is configured to sort the pixel data into odd row pixel data and even row pixel data, and wherein the FPA is further configured to sort the pixel data into odd column and even column odd row pixel data and odd column and even column even row pixel data.
 17. The image sensor of claim 15, wherein the FPA is configured to sort the pixel data into color categories with a Bayer color filter, the color categories comprising red pixel data, blue pixel data, a first portion of green pixel data, and a second portion of green pixel data.
 18. The image sensor of claim 15, further comprising one or more buffers coupled to the FPA configured to buffer pixel data from the FPA for one or more captured images prior to compression of the pixel data.
 19. The image sensor of claim 15, further comprising a plurality of dedicated pixel data feeds coupling the compression engine to the FPA, wherein each of the plurality of pixel data categories is associated with one of the dedicated pixel data feeds, and wherein pixel data associated with each of the plurality of pixel data categories is transmitted to the FPA via the dedicated pixel data feed associated with the pixel data category.
 20. The method of claim 1, wherein the data compression factor comprises a factor of approximately 3 or more.
 21. The method of claim 1, further comprising: receiving subsequent pixel data from the focal plane array at a second data rate greater than the first data rate; sorting the subsequent pixel data into the plurality of categories; selecting a second data compression factor based on the second data rate, the second data compression factor representing a greater magnitude of compression than the first data compression factor; and compressing the categorized subsequent pixel data in parallel based on the selected second data compression factor.
 22. The method of claim 1, further comprising: receiving subsequent pixel data from the focal plane array at a second data rate less than the first data rate; sorting the subsequent pixel data into the plurality of categories; selecting a second data compression factor based on the second data rate, the second data compression factor representing a lesser magnitude of compression than the first data compression factor; and compressing the categorized subsequent pixel data in parallel based on the selected second data compression factor.
 23. The method of claim 1, further comprising: determining an amount of power available to output pixel data; and selecting the first data compression factor based additionally on the determined amount of power available to output pixel data. 